Skip to content

Bound shutdown waits and retain Python cleanup - #25

Open
ChrisKrawczyk wants to merge 8 commits into
mainfrom
feature/rust-dotnet-lifecycle
Open

ChrisKrawczyk wants to merge 8 commits into
mainfrom
feature/rust-dotnet-lifecycle

Conversation

@ChrisKrawczyk

@ChrisKrawczyk ChrisKrawczyk commented Oct 6, 2026 •

Copy link
Copy Markdown

Summary

Bound Python shutdown waits without discarding native cleanup, and expose passive stop observation. Successful shutdown still returns None.

  • Use fallible core startup and retain prepared-core ownership before activation with the GIL released.
  • Validate durations before accepting stop. Repeated calls retain the first deadlines and observe the original operation.
  • Keep registry/preparation failures as sanitized RuntimeError values without changing registration policy; distinguish shutdown TimeoutError.
  • Release obsolete activity-callback ownership only after proven cleanup, retaining terminal results/deadlines separately.
  • Add synchronous runtime.stop_observer() metadata observation and isolated lifecycle/observation controls, repeated races and native provenance.

The latest commit, f427a32, updates three documents after real-binding qualification against the cleanup-policy implementation in microsoft/duroxide#64 at cc06a9e. No production Python correction, dependency change or new public API was needed for that increment. The earlier lifecycle/observation implementation and upstream reconciliation remain in this PR.

flowchart TD
    Stop["First valid stop"] --> Cleanup["Keep native cleanup owned"]
    Stop --> Wait["Wait within grace plus 5000 ms"]
    Cleanup -->|Cleanup proven| Release["Release obsolete core and activity-callback ownership"]
    Cleanup -->|Cleanup unproven| Retain["Retain ownership"]
    Wait -->|Total expires first| Timeout["Raise TimeoutError; keep cleanup owned"]
    Timeout --> Terminate["Application or supervisor must terminate the process"]
Loading

Behavior and review focus

Omitted grace remains 1000 ms, with a total of grace plus 5000 ms. Zero requests force, not instant cleanup. Pre-start shutdown is terminal. Serialize lifecycle/registration calls on each Runtime, including calls from other threads while the GIL is released. Metrics remain unavailable after stop or accepted startup failure.

Timeout still requires application- or supervisor-owned process termination, even after later cleanup. The SDK never exits the process itself. RuntimeError or lifecycle_shutdown_failed alone does not prove cleanup; owner loss still requires retained ownership. PyO3 defers off-GIL reference-count decrements until the next GIL entry. Independent providers and Clients keep their lifetimes, and the global orchestration-handler map is unchanged.

A synchronous Python callback stays owned until it returns, even if shutdown times out. Timeout/native drop cannot preempt it. Application-created threads and other unregistered work remain separately owned; this binding does not gain .NET's foreign-completion barrier.

The matching core contains only named user-cleanup failures. Eligible late destruction preserves actual outer outcomes or valid finalized decisions, not an inner success followed by outer failure. Core, reporting, finalization and registered-cleanup failures remain fatal. Valid output/decisions do not establish provider commit.

Contained incidents use native tracing target duroxide::runtime::cleanup, category contained_user_cleanup; they do not themselves set Faulted. Stop observation is not an incident feed or a health check. At most two protected payload-disposal attempts can still block; cleanup-skipped means an allocation was left undisposed. Containment does not repair state or recover process abort. Returned lifecycle errors exclude panic payloads, but the separate process-wide Rust panic hook can print them.

stop_observer() retains metadata rather than Runtime/provider/registry ownership. Read state before synchronous wait_for_change(last_seen, timeout_ms=None). The wait releases the GIL and checks Python signals; timeout and KeyboardInterrupt do not request shutdown. Each observer permits one pending wait. Closing it from another thread wakes the wait and reports observer closure. Source closure delivers an unseen final state before None; neither closure nor a status proves cleanup. Async callers must offload blocking waits and arrange finite waits/closure rather than assuming asyncio cancellation cancels native observation.

The lifecycle guide
documents driver preconditions, observation and ownership. The latest documentation increment preserves executable examples and diagrams.

Current cleanup-policy qualification: November 18-20

Two fresh Windows x64 development/unwind native builds used the actual maturin path and separate Python 3.12 environments with the reviewed core implementation.

Python selection Result
Instrumented real-binding probes 7 passed
Production-feature public controls and loaded-native identity 8 passed

Compiled features and loaded extension hashes were checked; production test-only exports are absent. Probes cover contained versus fatal failures, actual generator disposal/valid decisions, inner-success/outer-failure behavior, synchronous callbacks and separately created application threads.
The held synchronous callback stayed owned through a measured 5.016-second zero-grace shutdown timeout; the original 5 <= elapsedSeconds < 6 assertion was unchanged. Existing missing-token cancellation behavior remains false. No budget or watchdog was relaxed.

The source-built graph uses modified core declaring 0.1.32, Tokio 1.54.1, futures-util 0.3.34 and resolved PG 0.1.38; the manifest declares PG 0.1.37. Original pins, locks and native assets are unchanged. Published core 0.1.32 is not equivalent to that modified source.
The full cross-SDK selection passed 109 distinct cases / 112 executions across six native builds, with no final failures, skips, zero selections, aborts or watchdog timeouts. No production SDK correction was needed. This is local SQLite/in-memory development evidence, not Python 3.13, optimized, PostgreSQL, platform or installed-package qualification of the new policy.

The current core selection passed 372 distinct cases / 712 executions. The documentation audit and independent review/challenge found no blocking defect; optional Runtime-lifetime correlation for cleanup incidents was not added.

Earlier evidence and remaining gates

Before the new policy, reconciliation checks on Python 3.12 and 3.13 each passed 30 hook-free lifecycle cases with 9 expected hook-only skips and 39 instrumented cases. Python 3.12 additionally passed 42 PostgreSQL-backed E2E cases including the SQLite control; instrumented all-target Clippy passed. Both ordered races retained 100 iterations. That earlier graph used registry PG 0.1.37. Those dev builds and PostgreSQL results are historical, not reruns of this policy. The temporary database container and credentials were removed afterward.
Later passive-observation evidence is summarized in microsoft/duroxide#64, separately from the fresh cleanup-policy matrix.

Depends on microsoft/duroxide#64. The earlier hosted registry failure records the missing published lifecycle API dependency. Current-head CI must be assessed separately.
Compatible core publication under maintainer authorization, final registry minima/provider graph/locks/native assets, and hosted, optimized/release-profile, supported-platform, PostgreSQL and installed-package checks remain gates. A provider version alone does not qualify a shared production database migration.

No release, package publication, publishing dispatch or merge into main was performed or requested. This delivery does not change dependency overrides or locks, and commits no native extensions, workflow artifacts or unrelated work.

Generated with PAW.

chkraw and others added 2 commits October 6, 2026 14:33
Preserve the prepared core and startup failures across repeated lifecycle
calls, validate shutdown deadlines before mutation, and distinguish timeout
from operational failure while retaining successful None return values.
Add supervised lifecycle regressions, test-hook wiring, and migration docs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 73748d4f-2eb6-47d2-9dbc-af8681e80e83
Preserve upstream replay isolation, failed-race semantics, trailing lock options, release metadata, and the Rust toolchain pin. Integrate options into protected startup preparation and add raw/typed lifecycle race regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 73748d4f-2eb6-47d2-9dbc-af8681e80e83
@ChrisKrawczyk

Copy link
Copy Markdown
Author

Conflict reconciliation is pushed in b8c3213 (a normal merge of main; no history rewrite). GitHub now reports this draft conflict-free. Upstream replay/race fixes, trailing lock-option parameters, release metadata and the toolchain pin are retained, with option preparation still inside the protected lifecycle startup path.

The requested independent Claude Opus 5 integration review covered all four lifecycle PRs and returned PASS, with no blocking findings. The reviewed tree is exactly the committed tree; microsoft/duroxide#64 is unchanged. Non-blocking follow-ups include choosing a new sibling package version before release and the existing signed-to-unsigned negative-option validation gap. This supersedes the original conflict note only: the published core/provider graph, final registry pins, PostgreSQL, non-Windows/five-RID, broader regressions and hosted CI remain unwaived gates. The PR stays draft; nothing was merged, released or published.

Retain one native cleanup operation independently of bounded callers and
preserve the first SDK budget, terminal state and Python error shapes.
Integrate upstream release/replay changes and exercise cancellation-safe
provider futures separately from non-yielding native work.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 73748d4f-2eb6-47d2-9dbc-af8681e80e83
@ChrisKrawczyk

Copy link
Copy Markdown
Author

Upstream-first reconciliation is pushed in beec0e7 (normal merge of main; no history rewrite). GitHub reports this draft conflict-free. The adapter retains one native cleanup operation and separately bounds repeated observations against the first SDK budget, preserving successful None returns and Python error shapes. Upstream 0.2.1 release metadata and replay/KV fixes are retained. Tests distinguish cancellable fetch futures from non-yielding native frames; no guarantee for unregistered Python work was added.

The independent scoped four-repository review is PASS, including the corrected inherited .NET concurrency-test defect. Python's instrumented lifecycle suite passed 20 cases, including 100 repetitions per force ordering and isolated timeout/late-retirement probes. Production lifecycle/lock/replay/typed selections passed 26 with nine expected hook/POSIX skips. Production assets are restored; the reviewed source tree exactly matches the commit.

Core microsoft/duroxide#64 now uses upstream task tracking and is about 32% smaller by feature changed-line count. The local core 0.1.32/provider 0.1.36 graph is development-only: published core lacks additive APIs. Registry delivery, PostgreSQL, non-Windows/five-RID, broader regression and hosted CI remain OPEN. The PR stays draft; nothing was merged, released or published.

chkraw and others added 3 commits October 21, 2026 13:57
…rage

Separate retained shutdown results from callback-bearing ownership. Prove collection after quiescence and strengthen startup, registration, timing and subprocess diagnostic regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 73748d4f-2eb6-47d2-9dbc-af8681e80e83
Keep lifecycle changes under Unreleased while adopting main version metadata and duroxide-pg 0.1.37. Verify both native modes on Python 3.12 and 3.13 against the retained local-core graph; compatible core publication remains a separate gate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 73748d4f-2eb6-47d2-9dbc-af8681e80e83
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 73748d4f-2eb6-47d2-9dbc-af8681e80e83
@ChrisKrawczyk ChrisKrawczyk changed the title [Sibling Runtime Lifecycle] Add retained bounded shutdown for Python Bound shutdown waits and retain Python cleanup Oct 21, 2026
@ChrisKrawczyk
ChrisKrawczyk marked this pull request as ready for review October 22, 2026 04:55
chkraw and others added 2 commits November 4, 2026 17:25
Expose string-valued stop state and independently owned synchronous observers. Release the GIL during waits, preserve finite timeout deadlines, propagate signals and support cross-thread close without worker shutdown. Cover real faults, collection and notification lifetime, and document blocking/asyncio boundaries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 73748d4f-2eb6-47d2-9dbc-af8681e80e83
Explain the qualified core cleanup policy and synchronous callback
ownership separately from detached work and passive stop observation.
Preserve existing shutdown budgets and interruption semantics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 73748d4f-2eb6-47d2-9dbc-af8681e80e83

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants